Skip to content

[FLINK-38866][flink-table-planner] Passing timestamp precision from window strategy to window operators to support non-compact BinaryRowData timestamps #27386

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
mattcuento wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[FLINK-38866][flink-table-planner] Passing timestamp precision from window strategy to window operators to support non-compact BinaryRowData timestamps #27386

mattcuento wants to merge 1 commit into from
+520 −7

Conversation

@mattcuento
Copy link

@mattcuento mattcuento commented Jan 5, 2026 •
edited
Loading

What is the purpose of the change

BinaryRowData stores timestamps in a non-compact form when precision is > 3. Currently, window operator code hardcodes precision to 3. So, when reading back a non-compact timestamp, incorrect deserialization occurs resulting in incorrect windows.

Users are only exposed to this bug when using BinaryRowData with a descriptor that has precision > 3 and is running the job in batch mode.

In streaming mode, descriptors must be event time attributes, which must be a precision of 3 at maximum by default.

Brief change log

  • Extract the timestamp precision from the window assigner in StreamExecWindowTableFunction to pass to UnalignedWindowTableFunctionOperator
  • Extract the timestamp precision from the window assigner in CommonExecWindowTableFunction to pass to AlignedWindowTableFunctionOperator
  • Extend WindowTableFunctionOperatorBase constructor to take in timestampPrecision as an int in a protected field
  • Extend AlignedWindowTableFunctionOperator and UnalignedWindowTableFunctionOperator constructors to take in timestampPrecision int argument, use it in processElement for dynamically handling non-compact BinaryRowData

Verifying this change

Please make sure both new and modified tests in this PR follow the conventions for tests defined in our code quality guide.

This change added tests and can be verified as follows:

  • Added an end-to-end Batch SQL test with non-compact/compact timestamp representations on a query that forces use of BinaryRowData

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (not applicable)

@flinkbot
Copy link
Collaborator

flinkbot commented Jan 5, 2026 •
edited
Loading

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@mattcuento mattcuento changed the title [FLINK-38866][flink-table-planner] Validating field precision for window descriptors [FLINK-38866][flink-table-planner] Passing timestamp precision from window strategy to window operators to support non-compact BinaryRowData timestamps Jan 6, 2026
davidradl
davidradl reviewed Jan 6, 2026
View reviewed changes
...he/flink/table/runtime/operators/window/tvf/operator/AlignedWindowTableFunctionOperator.java
int timestampPrecision,
ZoneId shiftTimeZone) {
super(windowAssigner, rowtimeIndex, shiftTimeZone);
super(windowAssigner, rowtimeIndex, timestampPrecision, shiftTimeZone);
Copy link
Contributor

@davidradl davidradl Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be cleaner to keep the original method signature, and call the super(windowAssigner, rowtimeIndex,shiftTimeZone); The default then live in the WindowTableFunctionOperatorBase, rather than being specified in all the callers

The callers that need to pass the precision can use the new method.

Copy link
Author

@mattcuento mattcuento Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might be a bit confused on what you mean. Is the desire for a new constructor in WindowTableFunctionOperatorBase with a signature that passes timestampPrecision?

If so, the only two usages are for these two implementations, AlignedWindowTableFunctionOperator and UnalignedWindowTableFunctionOperator. Intended behavior here would be to always pass timestampPrecision through a concrete operator instance as opposed to defaulting to any given value

@github-actions github-actions bot added the community-reviewed PR has been reviewed by the community. label Jan 6, 2026
@mattcuento
[FLINK-38866][flink-table-planner] Passing timestamp precision to win…
32ebe4a 
…dow operators to respect BinaryRowData non-compact timestamp form

[FLINK-38866][flink-table-planner] Validating field precision for window descriptors

Revert validation

[FLINK-38866][flink-table-planner] Passing timestamp precision to window operators to respect BinaryRowData non-compact timestamp form

Add test

Add test

Add test

Add test
@mattcuento
Copy link
Author

mattcuento commented Jan 12, 2026

@flinkbot run azure

1 similar comment
@mattcuento
Copy link
Author

mattcuento commented Jan 13, 2026

@flinkbot run azure

@mattcuento mattcuento requested a review from davidradl January 13, 2026 17:22
@mattcuento
Copy link
Author

mattcuento commented Jan 13, 2026

Hey 👋 @davidradl , it appears there's a flaky pyflink test that seems unrelated to my change. (appears to be a failure on pip install for python and ERROR: failed to build: failed to solve: process "/bin/sh -c apt-get update && apt-get install build-essential -y" did not complete successfully: exit code: 100 for e2e).

Would you be able to trigger a re-run/ have you encountered this failure before?

I've tried @flinkbot run azure but it appears results are cached by commit hash and I don't have permission to re-run via Azure UI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@davidradl davidradl Awaiting requested review from davidradl

Assignees

No one assigned

Labels

community-reviewed PR has been reviewed by the community.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mattcuento @flinkbot @davidradl